Subject: TCP connection management

From: [email-masked]
Date: Wed, 11 Aug 93 21:13:00 EDT

Andy, sorry about the delay in responding....

Susie Kim Riley correctly extrapolated my earlier append about the
dual TCP connections, in choosing the second of the possibilities
you presented.  The 6611 connects out to its DLSw partners on port
2067, and listens to and accepts connections on port 2065.  After
accepting a connection, we connect out to the partner who just
connected to us, if we haven't already done so.  Partner definition
is by sysdef, but only one of each pair of routers need be configured
to the other.

The dual TCP connection is not necessary for data flow since of course
a single connection is full-duplex.  The configuration proved to be
the best approach for the 6611 at the time it was being developed, but
may not be best for our SIG standard to settle on (provided we address
6611 compatibility).  As we consider the alternatives (note that Scott
Adams' cap exch suggestion today suggests a range), let me mention a few
scattered points:
  - Several times people have proposed multi-connection capability
    to relieve possible network congestion caused by many circuits
    following the same path.  Depending on the forwarding logic being
    used within the network, additional connections which have the
    same destination IP address or (source IP, destination IP) will
    follow the same path through the network.  Note that each of the
    current pair of TCP connections may actually follow different
    paths in some scenarios! (not why it was done, to my knowledge)
  - We should consider the customer impact of having to define new
    IP address relationships, even at a central location.  Our
    experience is that this task is difficult for users, even with
    a nice configure utility.
  - One possibility for reducing the requirement to one TCP connection
    would be to bring up two as today and negotiate to drop one.  In
    no case should we require partner sysdef at both nodes or
    pre-definition of which partner should initiate the connection.

I think the idea of one TCP connection per circuit has a number of
problems.
  - Resources for all the TCP connections, both memory and processing
    load.  Consider the need for thousands of connections in a single
    router.  Regardless of 1 versus 2 TCP connections, Marcia's point
    about scalability is valid - TCP takes more resources per connection
    than I think any reasonable DLSw implementation would take per
    circuit.
  - There are lots of timing issues bringing up circuits; adding the
    delays of bringing up a TCP connection further complicates the
    problems.  DLSw is simple conceptually, but getting it all to
    really work is not easy!
And I have two questions:
  - How will we do addressing between all these parallel TCP connec-
    tions?  Does the source TCP port number need to become a number
    with only local significance?  Note again that this won't affect
    the network path that the traffic takes.
  - Andy, you mentioned we might go to one TCP connection "if traffic
    is going to be heavy".  Do you have a proposal for how a router
    will know at circuit establishment time what the traffic character-
    istics will be?  Is this something a customer would sysdef for
    certain pairs of end station addresses?

Regards,
Peter

---

From: [email-masked] (Susie Kim Riley)
Date: Thu, 12 Aug 93 08:56:48 EDT

>>    pre-definition of which partner should initiate the connection.

This approach seems to make the most sense so far.  It will enable us
to interoperate with existing 6611s which use the simplex mode of
communication, as well as negotiate down to one duplex tcp connection
if the partner is capable.  We are going to have to handle the case of
two connections getting established at the same time anyway (if both
sides try to connect to each other at the same time).

>>    certain pairs of end station addresses?

We should be able to dynamically change the transport (i.e., tcp
connection) the DLS is using for a particular LLC/SDLC session.  We
could take note of how many times the transport connection gets into a
'congested' state.  If this happens frequently enough (determined by
some threshold we set ourselves), we open a new connection to the
other side, and rebind the sessions to the new transport connection.
Of course, this would not really help matters if the reason for the
congestion is a slow/bad network as opposed to too many sessions going
over one transport (tcp) connection.

SKR

---

From: [email-masked] (Anand Subramaniam)
Date: Thu, 12 Aug 93 15:24:06 PDT

Peter,

Thanks for the reply. I shall try my best to address some of the issues raised
by Peter and Marcia and also try to propose 2 models by which we can handle
the TCP connections. We'll let the group decide based on simplicity of the
model and ease of implementation.  If the group decides to accept one of
these, then I'll come out with all the technical specification.

Model 1
=======

This is a one-one model. We'll have a transport connection per network
connection. 

Description:
------------

There will be only one listener port(DLSW_LISTEN). 

Connection Establishment
------------------------
Whenever the local DLSw wants to establish a DL connection, it issues a
transport connect to the peer. The peer accepts the connection the DLSw
can start its FSM. In this case the initiating DLSw is the "owner" of the
connection.

So the connection would look like

	X : IP-A :: DLSW_LISTEN : IP-B

X is a variable number and has only local significance. When IP-B initiates
the transport connection we have,

	X : IP-B :: DLSW_LISTEN : IP-A

Now for the advantages and disadvantages:

Advantages first,

1.	Once the transport connection is set up, we have the complete
	transport connection bandwidth for the DL connection.

2.	Transport itself provides a very comprehensive flow control mechanism.
	So the DLSw does not need any explicit flow control. This means that
	we need not introduce any new messages thus making DLSw more complex.
	Since the primary goal is to make DLSw simple, we shouldn't be making
	it more complex.

3.	Once the DL connection is in place, the data transfer will be smooth.
	The DL connection will have a better view of what is going on and no
	"bad neighbor" can affect this connection. 

4.	The going down of a particular transport connection will only affect
	the DL connection and not others.

5.	Scheduling DL connections becomes relatively easy because when you
	schedule a transport connection, you can schedule the DL connection
	associated with it. You don't have to have a complicated scheduling
	mechanism to allocate the transport bandwidth if we were to multiplex
	many DL connections.

6.	Since TCP is stream-oriented and not a packet-oriented protocol, we
	have usage of data buffers. 

7.	Since there is one-one correspondence any congestion on the TCP side
	will get properly reflected on the DL side.

Disadvantages, next

1.	More TCP connections means more control blocks per TCP connection 
	which means more memory. I see control blocks as ones that store
	transport related information and should not consume a lot of memory.
	If we need to support a LOT of DL connections, then we should not
	really worry about memory. Assuming the control blocks are 256 bytes
	each , for 2000 TCP connections we have, 512 K of extra memory. If we
	were to use the pacing scheme, we might have to have pre-allocated
	buffers in the DL connection which might end up with the same number.

2.	One thing I am worried is about certain implementations out there
	which have pre-allocated buffers, deterministic number of connections
	etc where there are a lot of constraints...

Issues
======

Memory and Processing Load (Peter)
--------------------------

As far as I can see, the only memory that is used will be the protocol control
block associated with the TCP connection. Ofcourse there might be a number of
implementations out there which have pre-allocated send and receive buffers
per TCP connections .... and every implementation  is unique in one way or
another. 

As far as the TCP connection establishment time, it is usually small.  Once
the connection is established, data would be streaming. I believe that we must
attempt to provide efficient and fast data paths for the data link
connections. Image the time we would be spending because of pacing , windows
being full, sharing of TCP bandwidth with the DL connections.. No scheduling
algorithm can be very fair.

Responses to Peter's question
=============================

1.	Addressing between TCP connections

	Yes, the source port has only local significance really. The
	connection is initiated based on the destination port and the local
	TCP(normally) would allocate port numbers from 1024 onwards.

2.	We might go to one TCP connection if traffic is heavy ,

	No. In my mail dated Aug 10(in response to Choon Lee), I mentioned
	that multiplexing class should be used when the traffic is not heavy
	on the transport connection.

Model 2
=======

In this model, we use the existing scheme in the RFC but enhance it so that it
fits in well with the flow control scheme. We have a TCP connection pair(no
doubts about this anymore!). 

In my earlier mail I had discussed about using the "unused" channels in the
connection pair. We could use these channels for all other messages other than
data so that 2 channels are used for data and the other for control.

So we have,
		------------->
	2067	<++++++++++++	2065

	IP-A			IP-B
		<-------------
	2065	+++++++++++++>	2067

	-----	indicates the info channel
	+++++	indicates the control channel.

We could transmit the CONTACT, CONTACTED, IPMs, DISCONNECT .....on the control
channel. This way we don't deviate from the RFC. Alternately we could define 2
ports 	2065 for DLSW_CONTROL and 2067 for DLSW_DATA. This is a minor issue.

Advantages, first
-----------------

1.	This fits in well with the existing implementations. 

2.	It gives a greater bandwidth for DL connections. We won't have the
	problem  of a congested TCP which has a lot of DISCONNECTS stuck in
	between the data and even if the DL connections want to break the
	connections and release bandwidth, the peer would not know.

3.	IPMs would reach the peer faster now and the DLSw can flow control
	better. 

4.	Fewer TCP resources used.

Disadvantages, next
-------------------

1.	We need to have a complicated flow control scheme in the DLSw. 

2.	When we multiplex connections, every implementation has to implement a
	fair bandwidth share alogorithm(on what basis?).

3.	Redundancy in flow control since TCP already provides a good one.	

4.	We cannot afford to have a closed TCP window as this will hose
	everyone else. In implementations which have pre-allocated buffers,
	this could easily happen and imagine around 2000 connections going
	down. So TCP window always needs to be open.

5.	Some implementations have tied the TCP receive window with the TCP
	user's receive capability since there is no point in receiving more
	than you can swallow. 

Conclusion
==========

Each scheme has its own advantages and disadvantages. Personally, I like the
first scheme better because it does not entail  new schemes and messages  and
protocol validation .... and all the others things that come with new designs.
I would like to make DLSw as simple as possible so that we can start
implementing and deploying these soon. I have only attempted to highlight the
schemes and once we decide upon the road we are going to take, we'll make it
more technical.

I solicit your comments and criticisms.

Thanks

--andy
-- 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UUCP:  {ames,apple,sun,mtxinu}!excelan!andy         (Anand Subramaniam) |
| INTERNET:  [email-masked]                          Ph : (408)-473-8413 |
| " Now and then I go off in to my own little world; But they know me     |
|   there"                                                                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+